Reducing noise in digital images

ABSTRACT

A target digital image is received from an image sensor. The image is contaminated by noise of unknown magnitude that is represented by a reference digital image. A process is applied that uses statistical analysis of the target digital image and of the reference digital image to estimate a magnitude of the noise for at least some pixels of the target digital image.

This description relates to reducing noise in digital images.

When features that appear in digital images, for example in digitalimages produced by a CMOS image sensor, are to be to quantitativelyanalyzed (for example, in medical applications), it is useful to reduceor remove the noise from the images before the analysis. It also isuseful to reduce or remove the noise from the images in cases for whichthe signal that produces the digital images is low relative to thenoise.

One kind of noise in the pixel values that make up the image (calleddark current noise, see FIG. 1) represents random noise levels that areproduced by respective pixels of the CMOS sensor array whether or notlight is being received by the sensor. (We sometimes use the word pixelin two different senses, one to refer to the photosensitive element thatresides at a particular location on the sensor array and the other torefer to the picture element that resides at a particular location onthe image and has a value that corresponds to brightness. We sometimesuse the term digital image to refer to the array of pixel values thatmake up the image.) Aside from temperature, the physical characteristicsof each sensor pixel that govern its dark current level do not changeover time. The pixel's dark current level does, however, depend on thetemperature of the pixel. And the pixel's dark current causes charge tobuild up over time, so that the effect of dark current on a pixel valuedepends on the duration of exposure of the pixel.

Other artifacts in the images include vertical patterns (also calledfixed pattern noise), offset, and shot noise. Vertical patterns (seeFIG. 2) are due to unintended differences in the operations of therespective readout circuits of different columns of the array andgenerally do not change over time. Offset represents differences inoverall signal level (brightness) from image to image that result fromvariations in certain electrical properties of the readout circuitry.Each pixel value generated by the sensor array includes random shotnoise with variance proportional to the signal value.

Digital images produced by other kinds of image sensors can also besubject to dark current noise.

SUMMARY

In general, a target digital image is received from an image sensor. Theimage is contaminated by noise of unknown magnitude that is representedby a reference digital image. A process is applied that uses statisticalanalysis of the target digital image and of the reference digital imageto estimate a magnitude of the noise for at least some pixels of thetarget digital image.

Implementations may include one or more of the following features. Thesensor is a CMOS sensor or a CCD sensor. The noise comprises darkcurrent noise. The process estimates the dark current magnitude forevery pixel of the target digital image. The process comprises programinstructions. The statistical analysis includes a de-correlationanalysis with respect to the target digital image and the referencedigital image. The dark current magnitude estimates are produced withoutrequiring information about a temperature of the sensor or a duration ofexposure. The reference digital image is based on a dark current digitalimage that is substantially free of vertical patterns and has beengenerated from a grey digital image and a black digital image acquiredrespectively using different exposure periods. The reference digitalimage is based on a corrected dark current digital image that has beenprocessed to reduce the effect of low-frequency spatial trends acrossthe pixels of the CMOS sensor. The reference digital image is based on ade-correlation of the black digital image with the de-trended darkcurrent image. The process subtracts vertical patterns, pixel by pixel,from the target digital image to produce a vertical pattern correcteddigital image. The process applies a dark current removal function tothe vertical pattern corrected digital image to produce a dark currentcorrected digital image. The noise in every pixel of the target digitalimage is reduced using the estimated dark current levels. The processapplies an offset estimation and subtraction function to the darkcurrent corrected digital image to remove offset. The noise reducedtarget digital image is provided to a processor for use in analyzingfeatures of an image captured by the CMOS sensor. The target digitalimage includes possibly malignant lesions.

These and other features and aspects may be expressed as apparatus,methods, systems, program products, and in other ways.

Other advantages and features will become apparent from the followingdescription and from the claims.

DESCRIPTION

FIG. 1 is an image of dark current.

FIG. 2 is an image of vertical patterns.

FIGS. 3, 4A, and 4B are schematic flow diagrams.

FIGS. 5, 6, and 7 are images at stages of the calibration process.

FIGS. 8 and 9 are graphs.

As shown in FIG. 3, an input digital image 10 (which we call T, fortarget) generated by a CMOS sensor array 12 in response to light 14received through optics 20 from a target scene 16 (for example, skinwith a pigmented lesion 18) can be processed (after temporary storage instorage 38) by noise reduction software 22 (run by a processor 24) toproduce an output digital image 26 (which we call O) for use inquantitative analysis 28 (for example, to determine 30 whether thelesion is a malignant melanoma, using the MelaFind® melanoma detectionproduct of Electro-Optical Sciences, Inc., of Irvington, N.Y.).

Even though we describe an example in which the noise reduction isperformed on a digital image in the context of medical diagnosis, thenoise reduction process is applicable broadly to any digital imageproduced by a any image sensor array for any purpose and in any context,including any in which the noise-reduced digital image may be subjectedto later analysis for which noise in the digital image would beproblematic.

Although we also describe specific ways to reduce vertical patterns andoffset, we note that the technique for reducing dark current (and othernoise that can be characterized by a reference digital image) in thedigital image described here can be used in a wide variety ofapplications in the absence of vertical patterns and offset or in whichother kinds of noise may or may not be reduced and, if reduced, in whichthe reduction may or may not be done in the way described in the exampledescribed here.

Other kinds of processing may also be required with respect to digitalimages produced by sensors in various applications, including processingto correct optical effects associated with a specific lens andilluminator. The noise reduction techniques described here thus haveapplications not limited by any optical correction or optical correctionof a particular kind.

As shown in FIGS. 4A and 4B, prior to noise-reduction processing of theinput digital image, reference information is acquired and processed.

In one step in developing reference information, multiple independentsets of a grey digital image 34 are acquired in the dark at a known orunknown sensor temperature and stored in storage 38. The exposure timeis in an intermediate range to avoid saturation from long exposures andyet have a reliable measurement of dark current. FIG. 5 shows an exampleof a grey image acquired in the dark (intensities multiplied by 15 fordisplay purposes.)

The multiple grey digital images are averaged (39) to produce an averagegrey digital image 40 (which we call G) in which the level of shot noiseat the individual pixels is reduced.

Also prior to noise-reduction processing of the input digital image,multiple independent sets of a black digital image are acquired in thedark, if fixed pattern noise such as vertical patterns is present. Theexposure time for the black sets is shorter than for grey sets (forexample, as short an exposure time as the hardware permits) to have areliable measurement of the fixed pattern noise and to minimize the darkcurrent level in the black sets.

The multiple black images are averaged (39) to produce an average blackdigital image 44 (which we call B) in which the effect of shot noise atthe individual pixels is reduced, as it was for the average grey digitalimage.

The average black digital image B is subtracted (46) from the averagegray digital image G to produce a dark current digital image 48 (D). Thesubtraction of the black digital image from the gray digital imageproduces an image of pure dark current (free of vertical patterns). FIG.6 shows the image of FIG. 5 after subtraction of the vertical patterns(intensities multiplied by 15 for purposes of display).

Next, a de-trending function is applied (50) to the dark current digitalimage 48 to remove low-frequency spatial trends from the pixels of thedata 48, because the trend in the dark current digital image can becorrelated with the target digital image. This de-trending is done bysubdividing the entire array of the dark current digital image 48 intosub-arrays of N pixels by N pixels. Within each sub-array, the darkcurrent digital image values are fit to a quadratic function of twovariables, using a least-squares fit. The value of this quadraticfunction for each pixel is then subtracted from the actual dark currentvalue in that pixel. In practice, good results have been obtained withN=3. The result is a de-trended dark current digital image 52 (S).

Next, a pure vertical pattern digital image V 56 is generated by firstapplying a dark current removal function (54) to B. The removal functionin general returns a digital image that represents the difference, pixelby pixel, between (i) an input digital image B and (ii) a product of thedark current digital image D times a factor A1

V=B−A1*D,

where the de-correlation function finds a factor A1 with respect to twosets of digital image, B and S:

Correlation(B−A1*S,S)=0,

where the correlation is computed over all pixels in a specified regionof the image.

In other words, the de-correlation function determines a factor A1 thatde-correlates the digital image B from the digital image S over someregion of the image. With respect to the particular step 54 in FIG. 4,the de-correlation determines the magnitude (A1) of dark current D inthe black image B.

The de-correlation function is an example of a statistical analysis thatenables the dark current noise to be determined from target pixels andfrom reference pixels without the need to know the temperature of thesensor or the period of exposure. Other statistical approaches couldalso be used, such as a variance minimization analysis. The result ofstep 54 in the figure is the pure vertical pattern digital image V 56.

FIG. 7 shows the image of FIG. 6 after subtraction of dark current noise(intensities multiplied by 15 and 100 levels were added to each pixelfor display purposes).

FIG. 8 provides a graphical illustration of cross sections of the imagesof FIGS. 5, 6, and 7 without intensity adjustments. In FIG. 8, VRem isthe intensity after subtraction of fixed pattern noise; DCRem is theintensity after subtraction of fixed pattern noise and of dark current;OffRem is the intensity after subtraction of fixed pattern noise, ofdark current, and of offset. In the original image of FIG. 5, shot noiseis about 7-8 levels, and it is not removed by the calibration process.Reduction of the shot noise would require either spatial or temporalaveraging of images.

The steps described above need only be performed once, e.g., duringfactory calibration, and the resulting calibration digital images can bestored and used for a large number of target images over a long periodof time. It is not necessary to develop the calibration imageinformation again each time a target image is captured.

As shown in FIG. 4B, to reduce noise in T, the pure vertical patterndigital image V is subtracted (58), pixel by pixel, from T to yield avertical pattern corrected digital image T1 60:

T1=T−V.

Next, using the de-correlation function (54) over some region of theimage, the magnitude A2 of dark current D in that region of the image T1is determined from

Correlation(T1−A2*S,S)=0,

and then the dark current is removed from T1 to produce a dark currentcorrected digital image T2 64:

T2=T1−A2*D.

The CMOS sensor can be arranged to have a black region (for example, ina corner or along one of the edges) of the array which is screened fromlight (including any light from a target). The digital image from theblack region can be used to correct for offset in the image. The blackregion digital image is first processed by the vertical pattern removaland dark current removal steps described earlier and the resultingprocessed black region data are averaged 66 to produce an average blackregion value NB 68. The average black region value is subtracted 70 fromevery pixel of the dark current corrected digital image T2 of an imageto eliminate offset from the target image. The resulting offsetcorrected digital image T3 72 can be subjected to additional processingdepending on the circumstances.

For example, if it is of interest to remove artifacts from the digitalimage T3 due to non-uniformities imparted by the imaging system (such asnon-uniform target illumination or non-uniform sensor response) or todetermine actual reflectances in the digital image T3, a calibrationdigital image W may be acquired by imaging a uniform white target with aknown diffuse reflectance. This white target digital image is thensubjected to a series of operations that include vertical patternremoval (subtracting V, if applicable), determination of the magnitudeof dark current by applying de-correlation to W−V and S, dark currentremoval, and offset removal, to produce W1. A reflectance calibration 78may be applied to the digital image T3 to produce a reflectance digitalimage T4 80 by the following computation (performed pixel by pixel):

T4=(T3/W)*(E(W)/E(T3))*ρ

in which E is the exposure time, and ρ is reflectance of a whitecalibration target. The reflectance calibrations 78 removes from thedigital images T3 non-uniformities imparted by the imaging system.

Additional processing as needed can be performed on T4 to yield theoutput image O.

The process described above assumes that the temperature of the sensoris uniform at all locations across the sensor array. To accommodate thefact that the temperature may vary across the sensor, dark current couldbe estimated independently at different parts of the sensor and theindependent estimates applied separately to the corresponding portionsof the input digital image.

The process described above can be applied to monochromatic digitalimages provided by a sensor. In some examples, the process can beapplied to multiple digital images in different spectral ranges that areproduced simultaneously by the sensor (e.g., red, green, and blue—RGB).In such cases, the digital images in different spectral ranges may beprocessed independently as described above.

The process may take advantage of a statistical analysis to reduce theneed, for some sensors, to control the temperature or duration ofexposure as a way to reduce the effects of dark current noise.

The processes described above could be implemented in hardware,software, or firmware, or any combination of them.

Validation of the dark current estimation technique was performed bycomparing the dark current level retrieved from dark images speciallytaken at various exposure times in a climate-controlled environmentagainst the dark current level predicted by the estimation techniquedescribed above. It was demonstrated that a de-correlation-basedestimator is able to predict accurately the actual dark current level inindividual images even in the presence of unavoidable shot noise, asillustrated in FIG. 9.

Other implementations are within the scope of the claims. For example, avariance minimization analysis could be substituted for thede-correlation analysis.

The techniques described here may be useful not only for sensorsoperating in the visible and infrared ranges but also for x-rays andpossibly ultrasound, that is, for any sensors for which removal of darkcurrent noise or effects similar to dark current noise would be useful.

Although the discussion above is directed to dark current noisecorrection, similar techniques could be applied in other contexts inwhich any noise for which a reference image is known or can be obtainedand in which the magnitude of the noise in the target digital image isunknown.

1. A method comprising: receiving, from an image sensor, a targetdigital image contaminated by noise of unknown magnitude that isrepresented by a reference digital image; and applying a process thatuses statistical analysis of the target digital image and of thereference digital image, to estimate a magnitude of the noise for atleast some pixels of the target digital image.
 2. The method of claim 1in which the sensor comprises a CMOS sensor.
 3. The method of claim 1 inwhich the sensor comprises a CCD sensor.
 4. The method of claim 1 inwhich the noise comprises dark current noise.
 5. The method of claim 4in which the process estimates the dark current magnitude for everypixel of the target digital image.
 6. The method of claim 1 in whichapplying the process comprises executing program instructions.
 7. Themethod of claim 1 in which the statistical analysis comprises ade-correlation analysis with respect to the target digital image and thereference digital image.
 8. The method of claim 4 in which the darkcurrent level estimates are produced without requiring information abouta temperature of the sensor or a duration of exposure.
 9. The method ofclaim 1 in which the process subtracts fixed vertical patterns, pixel bypixel, from the target digital image.
 10. The method of claim 4 in whichthe process applies a dark current removal function to the verticalpattern corrected digital image to produce a dark current correcteddigital image.
 11. The method of claim 1 in which the process applies aoffset estimation and subtraction function to the dark current correcteddigital image to remove offset.
 12. The method of claim 4 also includingreducing the noise in every pixel of the target digital image using theestimated dark current levels for every pixel.
 13. The method of claim 1also including providing the noise reduced target digital image to aprocessor for use in analyzing features of an image captured by thesensor.
 14. The method of claim 13 in which the images include possiblymalignant lesions.
 15. The method of claim 4 in which the referencedigital image is based on a dark current digital image that issubstantially free of vertical patterns and has been generated from agrey image digital image and a black image digital image acquiredrespectively using different exposure times.
 16. The method of claim 4in which the reference digital image is based on a corrected darkcurrent digital image that has been processed to reduce the effect oflow-frequency spatial trends across the pixels of the sensor.
 17. Themethod of claim 4 in which the reference data are based on ade-correlation of a black image digital image with a de-trended darkcurrent digital image.
 18. The method of claim 1 in which the imagesensor comprises a monochromatic sensor.
 19. The method of claim 1 inwhich the image sensor comprises a sensor that images in two or moredifferent spectral bands simultaneously.
 20. The method of claim 17 inwhich the image sensor images in red, green, and blue bands (RGB). 21.The method of claim 4 in which the reference image is obtained from thesensor without external light.
 22. The method of claim 1 in which theimage sensor produces images in the visible or infrared ranges.
 23. Themethod of claim 1 in which the image sensor produces images using X-rayor ultrasound radiation.
 24. An apparatus comprising a softwareprocessor configured to receive, from an image sensor, a target digitalimage contaminated by noise of unknown magnitude that is represented bya reference digital image; and apply a process that uses statisticalanalysis of the target digital image and of the reference digital image,to estimate a magnitude of the noise for at least some pixels of thetarget digital image.
 25. The apparatus of claim 24 also comprising adevice to analyze the target digital image, in which noise has beenreduced using the estimated dark current levels, to determine a presenceor absence of a possibly malignant lesion.
 26. An apparatus comprisingmeans for receiving, from an image sensor, a target digital imagecontaminated by noise of unknown magnitude that is represented by areference digital image; and means for applying a process that usesstatistical analysis of the target digital image and of the referencedigital image obtained, to estimate a magnitude of the noise for atleast some pixels of the target digital image.